Field of the Invention
The present invention relates to an image processing apparatus and method for predicting a camera work (motion of camera) from a moving image shot by an image capturing apparatus having an optical image stabilization function, and an image capturing apparatus.
Description of the Related Art
Recently, a technique of analyzing a camera work including changes in position and angle of a camera from a moving image and enhancing user friendliness by using this result is being widely applied. For example, an augmented reality (AR) technique of synthesizing a shot video and computer graphics (CG) is becoming popular, starting with a technique for smartphones. In particular, a markerless AR technique which implements CG synthesis by analyzing a camera work using natural features as input information has a wide application range and is very promising (see, for example, G. Klein, D. Murray, “Parallel Tracking and Mapping on a Camera Phone”, In Proc. ISMAR '09). The analysis of camera work is positioned as a core technique for attractive applications and products such as moving image stabilization, three dimensional information acquisition, and a camera work measurement/recording apparatus serving as a motion sensor using vision (see, for example, Japanese Patent Laid-Open No. 2007-215114). In this manner, the spread of various applied techniques and products is expected.
When a video is shot by a handheld compact image capturing device such as a smartphone, the video blurs. Conventionally, the AR technique is a broadcasting technique used in a shooting studio or the like, and is often used in combination with a special stable image capturing jig such as a crane or steady cam, so a blur hardly stands out. If the compact image capturing device is combined with such a special jig to solve the problem of the blur, this impairs portability which is a merit of the compact image capturing device. It readily occurs to solve the problem of the blur by adopting an optical image stabilization mechanism in the image capturing apparatus. The use of the optical image stabilization mechanism has merits: a blur that occurs during charge accumulation can be suppressed to improve the image quality, search for corresponding points or motion vectors can be facilitated in image analysis, and the image stabilization margin can be saved in image stabilization (see, for example, Japanese Patent No. 4446041).
However, when the optical image stabilization mechanism is introduced to suppress a blur, a motion shift between a shot video and CG stands out and accurate camera trajectory information cannot be obtained under a given condition in, for example, the AR application. Examples of the given condition are as follows: shooting is performed with a wide-angle lens; the size of a synthesized object is large with respect to the shooting angle of view; three dimensional camera work prediction and synthesis are performed; the motion is large; the display screen is large and the resolution is high; and these conditions co-occur. In these cases, camera work information based on a higher-accuracy camera model is required for faithful reproduction of the reality.
In such a case, for example, in the AR application, the motions of a shot video and CG do not coincide with each other and look tilted. In the moving image stabilization application, a shake remains. In three dimensional information acquisition, an error is generated in acquired three dimensional information. For the original purpose of camera work prediction, no accurate camera trajectory information can be obtained.
In this case, a complicated camera model which considers even optical image stabilization and has many variables may be prepared to predict a camera work by using an optimization technique such as model fitting to image information or information extracted from an image. However, this measure increases the number of prediction variables and exponentially increases the calculation amount of optimization calculation, impairing calculation convergence and processing stability to be impractical.